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ABSTR?»CT 

In 1988 a study was conducted to determine the 
validity of candidate teacher licensure examinations for use in 
Tennessee under the 1984 Comprehensive Education Reform Act. The 
Department of Education conducted a study to determine the validity 
of 11 previously unvalidated or extensively revised tests for 
certification and to make recommendations about the minimum 
qualifying scores for any of these tests found valid. 3ome of the 
tests were National Teacher Examinations specialty area tests 
developed or revised since 1984 — art education; music education; 
school guidance and counseling; school psychologist; and special 
education. An additional six tests, developed by a consortium of 
states in conjunction with the Educational Testing service (ETS) , 
were tested: earth/space science; health education; marketing and 
distributive education; psychology; teaching hearing impaired 
students; and teaching visually impaired students. A total of 270 
personnel from higher education institutions and local school 
districts reviewed each test as follows: (1) content review by 
teacher educators; (2) job relevance review by local educators; and 
(3) knowledge estimation review by members of both groups for 11 of 
the tests not previously reviewed in Tennessee. Seventeen educational 
and lay representatives further served as a standards Committee, 
using the information from the panels to make decisions on validity 
and mii;imum qualifying scores. The Committee recommended minimum 
qualifying scores for four tests, but did not recommend scores for 
eight tests lacking normative data. Minimum scores will be set when 
data are available. The committee further recommended that the impact 
of the tests on special groups of examinees, such as minorities, be 
monitored carefully. Five appendices contain recommended qualifying 
scores and summaries of data. (SLD) 
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Introduction 

Statutory requirements for initial licensure/certification and endorsement 
of public school personnel in Tennessee were established by the Comprehensive 
Education Reform Act of 1984. The legislation mandated that applicants for 
initial certification must present minimum qualifying scores on secured tests 
of communication skills, general knowledge, professional knowledge, and 
endcrsement area specializations. The act stipulated that the requirements 
would become effective July 1, 1984, or as soon thereafter as the tests could 
be validated and have minimum qualifying scores established. 

Two studies have been conducted to ascertain the validity of candidate 
teacher licensure examinations for use in Tennessee and the minimum qualifying 
scores for valid tests. A brief overview of the results and implementation of 
the recommendations from the first study in 1984 is presented. The recent 1988 
study is described in more detail because the latter study demonstrates the 
procedures used 'In both studies. 

Overview of 1984 Study 
A statewide study conducted in 1984 determined that the three NTE Core 
Battery tests and 23 of the 25 NTE Specialty Area tests were valid to use as 
initial certification tests in Tennessee. The study also provided the data to 
establish minimum qualifying scores for the valid tests. Upon completion of 
the study, the State Board of Education immediately established minimum score 
requirements for the NTE Core Battery covering corranuni cation skills, general 
knowledge, and professional knowledge. Subsequently, the State Board of 
Education instituted minimum score requirements as recommended for NTE 
Specialty Area tests that correspond to 14 endorsement areas. The tests used 
and recommended minimum scores from the 1984 study are presented in Appendix A. 




Objectives of the 1988 Study 
The study conducted for the Tennessee State Department of Education in 
1988 had two objectives* They were (1) to determine the validity of 11 
previously unvalidated or extensively revised ETS tests as measures of the 
knowledge and academic skills required for specific initial certification 
endorsements of public school personnel in Tennessee and (2) to formulate 
recommendations on minimum qualifying scores for any of these tests determined 
to be valid and three previously validated tests for use in Tennessee. The 
study was delimited to potential use of the ETS tests for initial certification 
endorsement rather than to select personnel for employment. 

Description of Candidate Tests 
The 14 tests addressed in the study are secured instruments that are 
available from Educational Testing Service, a private, ncn-profit testing 
organization. Eight NTE Specialty Area tests that have been developed or 
revised since 1984 were included in the study (Art Education, Biology, 
Chemistry, Music Education, Physics, School Guidance and Counseling, School 
Psychologist, and Special Education). A consortium of states with testing 
programs for teacher certification has been formed to work with ETS in the 
development of tests for specific specialized areas that are not included among 
the NTE Specialty Area tests. Six tests developed by the state-sponsored 
testing program (SSTP) consortium were available for the study (Ecirth/Space 
Science, Health Education, Marketing and Distributive Education, Psychology, 
Teaching Hearing Impaired Students, and Teaching Visually Impaired Students). 

Strategy 

The methodology employed in the study involved groups or panels of teacher 
education institutional personnel in the review of test content, local school 



district professional staff in the assessment of test item job relevance, and 
both types of personnel in the estimation of knowledge levels by test item 
among minimally qualified applicants for the respective endorsement areas. The 
content review and job relevance review functions were applied with only 11 of 
the 14 tests because the tests for Biology, Chemistry, and Physics had been 
previously validated for use in Tennessee. A separate committee was named to 
review the data analyses of the responses from these panels in order to make 
decisions on test validity and recommendations on minimum qualifying scores for 
valid tests. 

Participant Nomination and Selection 
The nomination of potential panel members was solicited by letter that 
specified the number of nominees to be identified for each area of 
specialisation. Nominations were requested from the chief academic 
administrators of the teacher education units in Tennessee with approved 
preparation programs for the applicable specializations. The superintendents 
of representative local school districts throughout the state were requested to 
make nominations. 

The selection of panel members was n-^pformed by the study staff with 
attention being given to several factors, lo the extent possible within each 
area of specialization, nominees were selected on the basis of requisite expert 
qualifications, representation of gender groups, inclusion of relevant racial 
groups, and representation of institutions offering specialized preparation 
programs in Tennessee. A total of 270 personnel from higher education 
institutions and local school districts participated in the study. 

Panel Functions 

A current form of each ETS test was supplied by Educational Testing 
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Service for review by the panel members selected for each test. The panel 
members worked independently in conducting the reviews based on instructions 
given by ETS personnel, who supervised the data collection sessions. One-day 
meetings to collect the data were held in Chattanooga, Knoxville, Johnson City, 
Memphis, Jackson, and Nashville, respectively, during the time period of April 
13-22, 1988. 

Each content review panel member representing a higher education 
institution in Tennessee performed three tasks. First, the panelist examined 
each item on the assigned test to judge whether or not at least 90% of the 
students completing the appropriate preparation program would have the 
opportunity to acquire the knowlege or academic skills to choose the correct 
response for the item. Second, the panelist made judgments about the 
congruence between the proportion of the test devoted to each topic and the 
empha'sis on the topic in the curriculum required for professional preparation. 
Third, the panelist indicated the degree to which the test as a whole was 
congruent with the total professional preparation program. 

Each job relevance review panel member from a local school district 
reviewed each test item on the assigned test to make judgments about the 
relevance of the knowledge or academic skills to competent performance as a 
beginning specialized practitioner in Tennessee. The relevance or each item 
was judged as Crucial, Important, Questionable, or Not Relevant. 

The knowledge estimation panel members for 11 tests were the individuals 
who performed the content review and job relevance assessments. The knowledge 
estimation panel for each of the tests that had already been validated was also 
composed of subject-matter specialists from higher education institutions and 
local school districts. At the test item level, each panelist made judgments 
about the difficulty of each item for persons who have minimum levels of 
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knowledge and academic skills necessary for conpetent performance as a 
beginning specialized practitioner in Tennessee. 

Standards Coiranittee 

A group of 17 educational and lay representatives was selected to serve as 
the Standards Committee for the study. The committee met in Nashville on 
August 10-11, 1988, to perform two tasks. First, the committee reviewed the 
data on appropriateness (content review and job relevance review) for each test 
to make decisions on the validity of the 11 ETS tests under consideration for 
use ih Tennessee. Second, the committee developed recommendations on minimum 
qualifying scores for the valid ETS tests based on a review of the knowledge 
estimation data and examinee performance data. 



Two types of information are summarized in this section of the paper. 
They are demographic data on panel participants and the Standards Committee and 
results of the panel fuiiCtions (content review, job relevance review, and 
knowledge estimation)^ 

Demographic Da ta 

The personnel who performed the three panel functions were described by 
gender and racial background. The th/ee panel groups were distributed on these 
variables as follows: 



Presentation of Data 



Content Review 



Job Relevance Review Knowledge Estimation 
(N=113) (N=270) 



(N=99) 



Ger.de r 



Male 

Female 

No Response 



62% 
36% 
2% 



24% 
76% 
0% 



48% 
51% 
1% 



Racial Group 



Black 
White 

No Response 



9% 
84% 
1% 



14% 
83% 
3% 



11% 
82% 
7% 
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The 17-member Standards Committee was a broadly representative group of 
educators and lay personnel from Tennessee. The committee was described on the 
variables of gender and racial background as follows: 



Gender 

Male 53% 
Female 47% 



Racial Group 

Black 35% 
White 65% 



Content Review 

The content review of the ETS tests was performed at thr^e levels: test 
item, test topic, and total test. Appendix B presents the results of the item 
review for each of the 11 tests. The data indicated that ovo.r 50% of che 
panelists who reviewed each test reported at least 87% of the items for all 
tests except Art Education are included in the curricula of the specialized 
preparation programs. While the percentages of items included are generally 
lower, the same patterns exist when the 60% and 70% criteria are applied to the 
responses of the panelists. 

Appendix C contains a derived index for each test that represents the 
degree of difference between the topical emphases of the test and the 
specialized preparatory curriculum. Based on an index of 0 (close similarity) 
to 100 (little similarity), the range of the index values was from 14.4 for the 
Teaching Visually Impaired Students test to 64.7 for the School Guidance and 
Counseling test. With the exception of the latter test, the index values for 
the remainder of the tests were less than 50. 

The data summarized in Appendix D represent the comparison of each total 
test and the overall related professional preparatory curriculum. The 
percentages of panelists who indicated close parallel or some difference were 
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relatively high for most tests. Only two tests as a whole were judged to be 
appreciably dissimilar to the related preparatory program. 

Job Relevance Review 

Appendix E presents the results of the job relevance ratings of test items 
for each ETS tesf by local school district personnel. Responses of Crucial or 
Important were defined as indicating relevance for items in this analysis. 
Over 50% of the panelists who reviewed each test indicated that more than 80% 
of the items were relevant to competent performance as a beginning specialized 
practitioner in Tennessee. Over 70% of the panelists for each test judged 
about 60% or more of the items as being job relevant. 

Knowledge Estimation 

The responses of panel members in performing the knowledge estimation 
function for each of the 14 tests were analyzed to derive estimated raw score 
means for minimally qualified certification applicants. Using conversion 
factors provided by ETS, the scaled score equivalent of the raw score mean was 
computed for each test. The results for the 1988 study are reported in Table 
1. 

The scaled scores for each ETS subject-matter test can vary from a low of 
250 to a high of 990, a 740-point difference between xhe lowest and highest 
scores possible. Scores cannot be compared directly among the subject-matter 
tests for two reasons. First, the tests are normed independently on different 
groups of examinees. Second, the standard errors of measurement, an index of 
the precision of test scenes, vary to a considerable degree among the tests. 

Standards Committee Actions 
In order to act formally as a committee, the Standards Committee elected a 
member to serve as chair during its deliberations. The decisions on test 
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Table 1 



Summary of Analyses on Estimated Scores for Minimally 
Qualified Certification Candidates by Test 



Test (Number Items Mean 

and Name) Total /Score Raw Score Scaled Score N 



11 


Music Education 


150/150 


64.641 


551.193 


19 


13 


Art Education 


150/U5 


62.296 


552.839 


21 


23 


Biology 


150/150 


69.456 


480.158 


24 


O A 


Chemistry 


120/120 


52.602 


458.556 


18 


26 


Physics 


120/119 


53.457 


547.478 


20 


27 


Teaching Hearing 
Impaired 
Students 


120/120 


66.175 


531.354 


18 


28 


Teaching Visually 
Impaired 
Students 


120/120 


72.359 


579.463 


19 


35 


Special Education 


150/148 


73.525 


543.612 


19 


39 


Psychology 


120/120 


57.533 


464.147 


18 


40 


School Psychologist 


135/122 


60.792 


511.004 


16 


42 


School Guidance 
and Counseling 


145/140 


73.553 


592.008 


18 


55 


Health Education 


120/119 


60.594 


491.914 


22 


56 


Marketing and 
Distributive 
Education 


120/120 


69.459 


556.902 


19 


57 


Earth/Space Science 


120/116 


56.892 


504.139 


19 
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validity and recommendations on minimum qualifying scores and related matters 
are reported below. 

Test Validity Decisions 

The Standards Committee was presented all data collected and analyzed in 
performing the .ontent review and job relevance review functions. (Knowledge 
estimation data were not released to the committee until the test validity 
decisions had been made.) The committee considered concurrently the content 
review results (test item, test topic, and total test levels) and job relevance 
review results for each test independently in making decisions on test 
validity. Utilizing this approach, the committee concluded that 9 of the 11 ETS 
tests for which decisions were required were valid to use in Tennessee as 
initial certification endorsement area tests (see Table 2). The two tests 
declared invalid for this purpose were the Art Education and Earth/Space 
Science tests that were judged to be insufficiently congruent in content with 
the related professional preparatory programs. 

Recommended Minimum Qualifying Scores 

The Standards Committee received the results of the knowledge estimation 
function for minimally qualified professional practitioners based on the 
judgments of the panel members who reviewed the 14 ETS tests. The data were 
presented for two sub-sets of tests - the five tests (four of these having been 
ji iged valid) with sufficient numbers of examinees to derive normative 
information and the nine tests (eight of these having been judged valid) 
without sufficient numbers of examinees to derive normative information. 

The data reported for the tests with normative information were the 
following: number of examinees; suuled score mean, standard deviation, and 
standard error of measurement; and knowledge estimation scaled score mean. In 
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Table 2 



Compilation of Committee Decisions on Validity of 
Selected ETS Subject -Matter Tests for Teacher 
Licensure Endorsements in Tennessee 1988 Study 



Test (Number and Name) Validity Decision 



11 


Music Education 


Valid 


13 


Art Education 


Not Valid 


27 


Teaching Hearing Impaired Students 


Valid 


28 


Teaching Visually Impaired Students 


Valvd 


35 


Special Education 


Valid 


39 


Psychology 


Valid 


40 


School Psychologist 


Valid 


42 


School Guidance and Counseling 


Valid 


55 


Health Education 


Valid 


56 


Marketing and Distributive Education 


Valid 


57 


Earth/Space Science 


Not Valid 



addition, the values for scaled score means minus 1, 2, 3, and 4 standard 
errors of measurement were derived. The information for the tests with 
relatively small numbers of examinees included the following data: number of 
examinees, raw score mean and standard deviation, scaled score mean and 
standard deviation, and knowledge estimation mean. Data were not available for 
any of the tests on score distributions of examinees classified by any variable 
(e.g. region, state, or racial group). 

After £ tho'^ough review of the examinee performance data, the committee 
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recommmended specific minimum qualifying scores for the four valid tests with 
normative data (Biology, Music Education, School Guidance and Counseling, and 
Special Education). Further, the committee recommended that all applicants for 
initial certification with endorsements in the eight areas corresponding to the 
valid tests without normative data should be required to submit scores on the 
appropriate tests without any minimum qualifying scores being established. The 
recommendations for the 12 valid tests are summarized in Table 3. 

Table 3 

Committee Recommendations on Minimum Qualifying Scores 
for Valid ETS Tests to Use in Tennessee - 1988 Study 



Test (Number and Name) Recommendation* 



11 


Music Education 




480 


23 


Biology 




420 


24 


Chemistry 


No 


minimum 


26 


Physics 


No 


minimum 


27 


Teaching Hearing Impaired Student 


No 


minimum 


28 


Teaching Visually Impaired Students 


No 


minimum 


35 


Special Education 




490 


39 


Psychology 


No 


minimum 


40 


School Psychologist 


No 


minimum 


42 


School Guidance and Counseling 




540 


55 


Health Education 


No 


minimum 


56 


Marketing and Distributive Education 


No 


minimum 



Note: No minimum - score submission required without minimum score 
established 
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Additional Recommendations 

The committee as a whole was concerned about the potential adverse impact 
of recommended minimum qualifying scores on legally protected minorities as 
well as the lack of sufficient examinee performance data to recommend miiiimum 
qualifying scores for several tests. In response to these concerns, the 
committee recommended to the State Board of Education the following: 

1# Monitor the performance of examinees on the four tests for which 
minimum qualifying scores are recommended in order to determine the 
impact on specia. ^^.oups of examinees because data are not 
available currently to make judgments on this matter. 
2. Collect data on the performance of examinees who are required to 
submit scores for the eight tests without minimum qualifying scores, 
these data to include racial group membership, until sufficient data 
are available to be considered for setting minimum qualifying scores. 
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Appendix A 



Recommended Minimum Qualifying Scores on Valid NTE Core 
Battery and Specialty Area Tests by Time Period - 1984 Study 



Test Minimum Qualifying Score by Time Period 

After 

1984-86 1986-87 1987-88 : 

Core Battery 
Communication Skills 
General Knowledge 
Professional Knowledge 

Specialty Area* 
(Test Number and Name) 



1984-86 


1986-87 


1987-88 


1987-88 


640 


644 


647 


651 


637 


640 


644 


647 


631 


635 


639 


643 



1 


Education in the Elementary School 


490 


540 


560 


5Qn 


?. 


Early Childhood Education 


490 


560 


580 


610 


3 


Biology and General Science 


520 


560 


580 


600 


4 


English Language and Literature 


480 


520 


540 


570 


5 


Industrial Arts Education 


550 


610 


630 


650 


6 


Mathematics 


520 


590 


610 


640 


7 


Chemistry, Physics, and General Science 
Social Studies 


500 


580 


600 


620 


8 


490 


540 


570 


590 


9 


Physical Education 


540 


610 


630 


660 


10 


Business Education 


530 


580 


600 


630 


11 


Music Education 


480 


520 


540 


570 


12 


Home Economics Education 


490 


510 


540 


560 


17 


French 


490 


570 


590 


610 


18 


German 


470 


500 


520 


550 


19 


Spanish 


480 


530 


560 


580 


20 


Introduction to the Teaching of Reading 


470 


510 


540 


560 


22 


Speech Communication 


480 


560 


590 


620 


30 


Reading Specialist 


480 


520 


550 


570 


31 


Media Specialist— Library and Audiovisual 
Services 


500 


520 


550 


570 


33 


Speech-Language Pathology 


510 


540 


570 


590 


34 


Audiology 


570 


590 


610 


630 


41 


Educational Administration and Supervision 530 


550 


580 


610 


42 


Guidance Counselor 


540 


580 


600 


630 



♦Scaled scores for the NTE Specialty Area Tests are reported as multiples of 
10. 
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Appendix B 



Summary of Data Analyses on Content Review Ratings 
of Test Items by Test* 



Percentage of College Personnel 
Who Rated Item Content 
No. As Appropriate for Curriculum 
Test {Number of Over Over Over Over Over 

and Name) Items 50% 60% 70% 80% 90% N 



11 


Music Education 


150 


87 


73 


65 


55 


36 


12 


13 


Art Education 


150 


75 


62 


48 


31 


12 


9 


23 


Biology 


150 


NA 


NA 


NA 


NA 


NA 


NA 


24 


Chemistry 


120 


NA 


NA 


NA 


NA 


NA 


NA 


26 


Physi cs 


120 


NA 


NA 


NA 


NA . 


NA 


NA 


27 


Teaching Hearing 
Impaired Students 


120 


99 


98 


89 


74 


45 


9 


28 


Teaching Visually 
Impaired Students 


120 


93 


93 


77 


77 


44 


6 


35 


Special Education 


150 


94 


90 


79 


57 


32 


10 


39 


Psychology 


120 


93 


86 


71 


44 


13 


9 


40 


School Psychologist 


135 


90 


79 


79 


64 


33 


7 


42 


School Guidance 
and Counseling 


145 


99 


98 


91 


81 


41 


9 


55 


Health Education 


120 


92 


87 


82 


73 


50 


11 


56 


Marketing and 
Distributive Education 


120 


94 


94 


78 


48 


15 


8 


57 


Earth/Space Science 


120 


93 


78 


62 


40 


18 


9 



♦Percentages may not equal 100% for a test due to omiss'ons. 
NA = Not Applicable 
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Appendix C 



Summary of Data Analyses on Comparisons of Curriculum 
Content and Test Topic Emphasis by Test 



Test (Number 
and Name) 


Difference in 
Relative Emphasis 
Index* 


N 


11 


Music Education 


18.4 


11 


13 


Art Education 


29.2 


9 


23 


Biology 


NA 


NA 


OA 


Chemistry 


NA 


NA 


26 


Physics 


NA 


NA 


27 


Teaching Hearing 
Impaired Students 


CO .O 


Q 
O 


28 


Teaching Visually 
Impaired Students 


14.4 


6 


35 


Special Education 


20.6 


9 


39 


Psychology 


37.6 


9 


40 


School Psychologist 


48.1 


7 


42 


School Guidance 
and Counseling 


64.7 


7 


55 


Health Education 


19.1 


11 


56 


Marketing and 
Distributive Education 


47.3 


8 


57 


Earth/Space Science 


48.1 


7 



*Index range: 0 to 100 

Low score indicates close similarity; high score indicates little similarity. 
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Appendix D 



Summary of Data Analyses on Comparisons of 
Curriculum Content and Test Content by Test 



Percentage of College Personnel 



Choosing Each Response Option* 
Close Some Mucli Little 

Test (Number Parallel Difference Difference Similarity 

and Name) % % % % 



11 Music Education 27 

13 Art Education 22 

23 Biology NA 

24 Chemistry NA 

26 Physics NA 

27 Teaching Hearing 13 

Impaired Students 

28 Teaching Visually 0 

Impaired Students 

35 Special Education 33 

39 Psychology 0 

40 School Psychologist 14 

42 School Guidance 29 
and Counseling 

55 Health Education 0 

56 Marketing and 13 

Distributive Education 

57 Earth/Space Science 0 



46 27 0 11 

56 22 0 9 

NA NA NA NA 

NA NA NA NA 

NA NA NA NA 

88 0 0 8 

50 33 0 6 

67 0 0 9 

56 33 11 9 

57 14 0 7 
57 0 0 7 

82 9 0 11 

63 13 0 8 

57 43 0 7 



♦Percentages may not equal 100% for a test due to omissions. 
NA = Not Applicable 
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Appendix E 



Summary of Data Analyses on Job Relevance Ratings 
of Tests Items by Test* 



No. 

Test (Number of 
and Name) Items 



Percentage of Public 
School Personnel Who Rated 
Item Content as Relevant to Job 
Over Over Over Over Over 
50% 60% 70% 80% 90% N 



11 


Music Education 


150 


83 


70 


59 


48 


20 


9 


13 


Ml C uQULatlUll 


IDU 


o^: 


/O 


67 


51 


37 


12 


23 


Ri nl nnv 




MA 




MA 


MA 

NA 


NA 


NA 


24 


Chemistry 


120 


NA 


NA 


NA 


NA 


NA 


NA 


26 


Physics 


120 


NA 


NA 


NA 


NA 


NA 


NA 


27 


Teaching Hearing 
Impaired Students 


120 


98 


89 


78 


59 


30 


9 


28 


Teaching Visually 
Impaired Students 


120 


96 


93 


86 


78 


66 


13 


35 


Special Education 


150 


90 


86 


78 


64 


44 


9 


39 


Psychology 


120 


95 


86 


78 


63 


32 


10 


40 


School Psychologist 


135 


87 


76 


61 


43 


19 


9 


42 


School Guidance 
and Counseling 


145 


96 


90 


75 


49 


20 


9 


55 


Health Education 


120 


87 


81 


72 


59 


48 


12 


56 


Marketing and 
Distributive Education 


120 


98 


96 


94 


88 


66 


11 


57 


Earth/Space Science 


120 


96 


90 


78 


58 


25 


10 



♦Percentages may not equal 100% for a test due to omissions. 
NA = Not Applicable 



